Hybrid input machine learning frameworks

ABSTRACT

There is a need for more accurate and more efficient hybrid-input prediction steps/operations. This need can be addressed by, for example, techniques for efficient joint processing of data objects. In one example, a method includes: processing an audio data object using an audio processing machine learning model to generate an audio-based feature data object, processing an acceleration data object using an acceleration processing machine learning model to generate an acceleration-based feature data object, processing the audio-based feature data object and the acceleration-based feature data object using an feature synthesis machine learning model in order to generate a hybrid-input prediction data object; and performing one or more prediction-based actions based at least in part on the hybrid-input prediction data object.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/073,638 filed Oct. 19, 2020, which claims priority to andthe benefit of U.S. Provisional Patent Application No. 63/010,177, filedon Apr. 15, 2020 and U.S. Provisional Patent Application No. 63/024,046,filed on May 13, 2020, each of which is incorporated by reference hereinin its entirety.

BACKGROUND

Various embodiments of the present invention address technicalchallenges related to generating hybrid-input prediction data objects byjoint processing of multiple data objects having different formats(e.g., different sensory formats, different sampling formats, and/or thelike) and disclose innovative techniques for improving efficiency and/orreliability of hybrid-input prediction systems.

BRIEF SUMMARY

In general, embodiments of the present invention provide methods,apparatuses, systems, computing devices, computing entities, and/or thelike for performing hybrid-input prediction steps/operations thatrequire processing data contained in multiple data objects. Variousembodiments of the present invention disclose techniques forconsolidating (e.g., combining, merging and/or the like) data from twoor more unrelated data objects and generating hybrid-input predictiondata objects.

In accordance with one aspect, a method for joint processing of an audiodata object and an acceleration data object is provided. In oneembodiment, the method comprises processing the audio data object usingan audio processing machine learning model to generate an audio-basedfeature data object, wherein: the audio processing machine learningmodel comprises: (i) an audio model fast Fourier transform (FFT) layerthat is configured to process the audio data object in order to generatean audio model FFT output and (ii) an audio model one-dimensionalconvolutional layer that is configured to process the audio model FFToutput to generate an audio model convolutional output, and theaudio-based feature data object is generated based at least in part onthe audio model convolutional output; processing the acceleration dataobject using an acceleration processing machine learning model togenerate an acceleration-based feature data object, wherein: theacceleration processing machine learning model comprises: (i) anacceleration model FFT layer that is configured to process theacceleration data object to generate an acceleration model FFT output,(ii) an acceleration model one-dimensional convolutional layer that isconfigured to process the acceleration model FFT output to generate anacceleration model convolutional output, and (iii) an acceleration modelup-sampling layer that is configured to process the acceleration modelconvolutional output to generate an acceleration model up-samplingoutput, and the acceleration-based feature data object is generatedbased at least in part on the acceleration model up-sampling output;processing the audio-based feature data object and theacceleration-based feature data object using an feature synthesismachine learning model in order to generate a hybrid-input predictiondata object; and performing one or more prediction-based actions basedat least in part on the hybrid-input prediction data object.

In accordance with another aspect, an apparatus for joint processing ofan audio data object and an acceleration data object is provided, theapparatus comprising at least one processor and at least one memoryincluding program code, the at least one memory and the program codeconfigured to, with the processor, cause the apparatus to at least:process the audio data object using an audio processing machine learningmodel to generate an audio-based feature data object, wherein: the audioprocessing machine learning model comprises: (i) an audio model fastFourier transform (FFT) layer that is configured to process the audiodata object in order to generate an audio model FFT output and (ii) anaudio model one-dimensional convolutional layer that is configured toprocess the audio model FFT output to generate an audio modelconvolutional output, and the audio-based feature data object isgenerated based at least in part on the audio model convolutionaloutput; process the acceleration data object using an accelerationprocessing machine learning model to generate an acceleration-basedfeature data object, wherein: the acceleration processing machinelearning model comprises: (i) an acceleration model FFT layer that isconfigured to process the acceleration data object to generate anacceleration model FFT output, (ii) an acceleration modelone-dimensional convolutional layer that is configured to process theacceleration model FFT output to generate an acceleration modelconvolutional output, and (iii) an acceleration model up-sampling layerthat is configured to process the acceleration model convolutionaloutput to generate an acceleration model up-sampling output, and theacceleration-based feature data object is generated based at least inpart on the acceleration model up-sampling output; process theaudio-based feature data object and the acceleration-based feature dataobject using a feature synthesis machine learning model in order togenerate a hybrid-input prediction data object; and perform one or moreprediction-based actions based at least in part on the hybrid-inputprediction data object.

In accordance with yet another aspect, a computer program product forjoint processing of an audio data object and an acceleration data objectis provided, the computer program product comprising at least onenon-transitory computer-readable storage medium having computer-readableprogram code portions stored therein, the computer-readable program codeportions configured to: process the audio data object using an audioprocessing machine learning model to generate an audio-based featuredata object, wherein: the audio processing machine learning modelcomprises: (i) an audio model fast Fourier transform (FFT) layer that isconfigured to process the audio data object in order to generate anaudio model FFT output and (ii) an audio model one-dimensionalconvolutional layer that is configured to process the audio model FFToutput to generate an audio model convolutional output, and theaudio-based feature data object is generated based at least in part onthe audio model convolutional output; process the acceleration dataobject using an acceleration processing machine learning model togenerate an acceleration-based feature data object, wherein: theacceleration processing machine learning model comprises: (i) anacceleration model FFT layer that is configured to process theacceleration data object to generate an acceleration model FFT output,(ii) an acceleration model one-dimensional convolutional layer that isconfigured to process the acceleration model FFT output to generate anacceleration model convolutional output, and (iii) an acceleration modelup-sampling layer that is configured to process the acceleration modelconvolutional output to generate an acceleration model up-samplingoutput, and the acceleration-based feature data object is generatedbased at least in part on the acceleration model up-sampling output;process the audio-based feature data object and the acceleration-basedfeature data object using a feature synthesis machine learning model inorder to generate a hybrid-input prediction data object; and perform oneor more prediction-based actions based at least in part on thehybrid-input prediction data object.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described the invention in general terms, reference will nowbe made to the accompanying drawings, which are not necessarily drawn toscale, and wherein:

FIG. 1 provides an exemplary overview of a system architecture that canbe used to practice embodiments of the present invention.

FIG. 2 provides an example hybrid-input predictive computing entity inaccordance with some embodiments discussed herein.

FIG. 3 provides an example client computing entity in accordance withsome embodiments discussed herein.

FIG. 4 provides an exemplary schematic of a system for performinghybrid-input prediction steps/operations and generating hybrid-inputdata objects in accordance with some embodiments discussed herein.

FIG. 5 provides an example acceleration processing machine learningmodel in accordance with some embodiments discussed herein.

FIG. 6 provides an example audio processing machine learning model inaccordance with some embodiments discussed herein.

FIG. 7 provides an example feature synthesis machine learning model inaccordance with some embodiments discussed herein.

FIG. 8 provides an example acceleration model dimension adjustment layerin accordance with some embodiments discussed herein.

FIG. 9 provides a flowchart of an example process for generating anaudio-based feature data object in accordance with some embodimentsdiscussed herein.

FIG. 10 provides a flowchart of an example process for generating anacceleration-based feature data object in accordance with someembodiments discussed herein.

FIG. 11 provides a flowchart of an example process for generating ahybrid-input prediction data object in accordance with some embodimentsdiscussed herein.

FIG. 12 provides a flowchart of an example process for generating anacceleration model dimension adjustment output in accordance with someembodiments discussed herein.

FIG. 13 provides an example graphical representation illustratingvarious predictive outputs generated by the example hybrid-inputprediction data system in accordance with some embodiments discussedherein.

FIG. 14 provides an operational example of generating alerts based atleast in part on corresponding hybrid-input predictive outputs inaccordance with some embodiments discussed herein.

DETAILED DESCRIPTION

Various embodiments of the present invention are described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all embodiments of the inventions are shown. Indeed, theseinventions may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. The term “or” is used herein in both the alternativeand conjunctive sense, unless otherwise indicated. The terms“illustrative” and “exemplary” are used to be examples with noindication of quality level. Like numbers refer to like elementsthroughout. Moreover, while certain embodiments of the present inventionare described with reference to predictive data analysis, one ofordinary skill in the art will recognize that the disclosed concepts canbe used to perform other types of data analysis.

I. Overview

Various embodiments of the present invention disclose techniques forperforming joint processing of audio data and acceleration data thatimproves the efficiency of performing the noted joint processing andreliability of the generated results. Typically, predictive dataanalysis is performed by processing large amounts of data which are notconfigured and/or optimized for performing steps/operations on data withdissimilar characteristics (e.g., two or more data objects each havingdifferent dimensions and/or describing diverse types of information).There is a need for improved systems and methods configured to processdata with dissimilar characteristics. Additionally, there is a need forimproving hybrid-input prediction steps/operations to utilize less dataand/or sparse data as input such that fewer computing resources areutilized overall. Thus, various embodiments of the present inventionimprove hybrid-input prediction steps/operations by utilizing less inputdata or sparse input data without information loss while working withinthe limitations of the available data. The inventors have confirmed, viaexperiments and theoretical calculations, that various embodiments ofthe disclosed techniques improve efficiency and accuracy of hybrid-inputprediction systems and predictive data analysis relative to variousstate-of-the-art solutions.

Various embodiments of the present invention utilize machine learningmodels configured to perform steps/operations (e.g., convolutionalsteps/operations, up-sampling steps/operations and/or the like) withrespect to two or more dissimilar data objects describing unrelated data(e.g., raw audio data and raw accelerometer data) in order to generatehybrid-input prediction data outputs/objects describing important/novelfeatures of the data contained in the two or more data objects. Themachine learning models may modify one or both data objects in order toperform subsequent mathematical/arithmetic steps/operations andimportant/novel features from the data in the two or more dataoutputs/objects. Accordingly, accurate predictions can be obtained wheresome of the available data is limited or sparse and/or where it may beundesirable to utilize more resources to obtain richer data due to alack of complexity in the underlying type of information. By correlatingunrelated data to extract important features using the methods describedherein, the resulting hybrid-input prediction data outputs/objectscontain rich information and lead to more accurate predictions which maybe utilized to generate user interface data (e.g., real-time alerts) foran end user.

Accordingly, by utilizing some or all of the innovative techniquesdisclosed herein for performing hybrid-input predictionsteps/operations, various embodiments of the present invention increaseefficiency of data processing and accuracy of predictions. In doing so,various embodiments of the present invention make substantial technicalcontributions to the field of predictive data analysis and substantiallyimprove state-of-the-art hybrid-input prediction systems.

II. Definitions of Certain Terms

The term “audio data object” may refer to a data object that describes aset of information (e.g., an input dataset) corresponding to sound suchas raw audio data (e.g., recorded audio data for a monitoredindividual). Raw audio data may comprise one or more sound waves. Eachsound wave comprises a wavelength oscillating at a given frequency for aduration of time. An audio data object may describe such raw audio datain a time domain representation or the frequency domain representation.Raw audio data may be sampled at a given sampling rate (e.g., 44.1 kHz)to generate a time domain representation. In general, a data objectgenerated based at least in part on a high sampling rate will generate adata object with more information/data than a data object generatedbased at least in part on a lower sampling rate. An example audio dataobject may be represented graphically by plotting extractedvalues/features as a function of time. An audio data object describingraw audio data in the time domain may comprise a two-dimensional matrixin which a first dimension corresponds with a number of segments in timeand a second dimension corresponds with a plurality of values/featuresbased at least in part on the frequencies of the sound waves occurringat corresponding segments in time.

The term “acceleration data object” may refer to a data object thatdescribes a set of information (e.g., an input dataset) corresponding toacceleration of motion of a body (e.g., a human) with respect to thebody's frame (e.g., raw acceleration data). An example acceleration dataobject may describe a body's change in orientation and/or movements(e.g., vibrations) in two or more directions (e.g., recorded cross-bodyacceleration data for a monitored individual). Raw acceleration data maybe represented graphically as a waveform plotted with respect to time inwhich positive values indicate an increase in velocity, negative valuesindicate a decrease in velocity and zero/null values indicate a constantvelocity. An acceleration data object may describe such raw accelerationdata in a time domain representation or the frequency domainrepresentation by sampling the raw acceleration data at a given samplingrate (e.g., 50 Hz). Sampling raw acceleration data at a high samplingrate (e.g., 44.1 kHz) is undesirable due to a lack of complexity in theunderlying data. An acceleration data object describing raw accelerationdata in the time domain may comprise a two-dimensional matrix having alength and a width in which a first dimension corresponds with a numberof segments in time and a second dimension corresponds with a pluralityof features associated with oscillations at different frequenciesoccurring at corresponding segments in time.

The term “audio processing machine learning model” may refer to a dataobject that describes parameters and/or hyper-parameters of machinelearning model configured to perform a plurality of steps/operationswith respect to an audio data object in order to generate audio-basedfeature data object. For example, the audio processing machine learningmodel may comprise a plurality of layers each configured to perform oneor more steps/operations with respect to an input data object (e.g.,audio data object). Each layer of the audio processing machine learningmodel may be configured to perform a plurality of steps/operations tomodify one or more dimensions of a corresponding input audio dataobject. An example audio processing machine learning model may compriseat least one audio model fast Fourier transform layer, and at least oneaudio model one-dimensional convolutional layer.

The term “audio model fast Fourier transform (FFT) layer” may refer to adata object that describes a layer of an audio processing machinelearning model configured to process an audio data object using an FFTfunction and generate an audio model FFT output. The audio model FFTlayer is configured to perform a plurality of steps/operations withrespect to an audio data object describing raw audio data in the timedomain in order to generate an audio data object describing raw audiodata in the frequency domain. The resulting audio model FFT output mayhave dimensions that are different from the dimensions of the input dataobject (e.g., audio data object). For example, the audio model FFT layermay apply a mask/filter to the input data object (e.g., audio dataobject) in order to extract relevant features and generate an audiomodel FFT output describing such features.

The term “audio model FFT output” may refer to a data object thatdescribes the output generated by an audio model FFT layer of an audioprocessing machine learning model. The audio model FFT output may referto a data object describing a two-dimensional matrix having a length anda width in which a first dimension corresponds with a plurality ofsegments in time and a second dimension corresponds with a plurality ofrelevant features.

The term “audio-based feature data object” may refer to a data objectthat is generated by an audio processing machine learning model and isconfigured to describe the output of processing particular audio data byan audio processing machine learning model. An example audio-basedfeature data object may comprise a two-dimensional matrix having alength and a width in which a first dimension corresponds with a numberof segments in time and a second dimension corresponds with a pluralityof features associated with frequencies of sound waves occurring at thecorresponding segments in time. The audio-based feature data object maycomprise dimensions that are different from the dimensions of the inputdata object (e.g., audio data object) processed by the machine learningmodel. For instance, the dimensions of the matrix of the audio-basedfeature data object may be truncated in the first dimension(x-direction) and lengthened in the second dimension (y-direction) suchthat the audio-based feature data object contains more informationsampled over fewer segments in time.

The term “audio model one-dimensional convolutional layer” may refer toa data object that describes layer of a machine learning modelconfigured to perform one or more convolutional steps/operations withrespect to an audio model FFT output and generate an audio modelconvolutional output. The audio model one-dimensional convolutionallayer is configured to extract feature data from an audio model FFToutput. For example, the audio model one-dimensional convolutional layermay extract feature data from an audio model FFT output in order togenerate an audio-based feature data object with different dimensionsfrom the audio model FFT output. An audio model convolutional output mayrefer to the output generated by an audio model one-dimensionalconvolutional layer of an audio processing machine learning model.

The term “acceleration processing machine learning model” may refer to adata object that describes parameters and/or hyper-parameters of machinelearning model configured to perform a plurality of steps/operationswith respect to an acceleration data object and generate anacceleration-based feature data object. For example, the accelerationprocessing machine learning model may comprise a plurality of layerseach configured to perform one or more steps/operations with respect toan input data object (e.g., acceleration data object). Each layer of theacceleration processing machine learning model may be configured toperform a plurality of steps/operations to modify one or more dimensionsof a corresponding input acceleration data object. An exampleacceleration processing machine learning model may comprise at least oneacceleration model fast Fourier transform layer, and at least oneacceleration model one-dimensional convolutional layer.

The term “acceleration-based feature data object” may refer to a dataobject that is generated by processing an acceleration data object usingan acceleration processing machine learning model. An exampleacceleration-based feature data object may comprise a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a number of segments in time and a second dimensioncorresponds with a plurality of features associated withoscillations/vibrations occurring at the corresponding segments in time.The acceleration-based feature data object may comprise dimensions thatare different from the dimensions of the input data object (e.g.,acceleration data object) processed by the machine learning model. Forinstance, the dimensions of the matrix of the acceleration-based featuredata object may be truncated in the first dimension (x-direction) andlengthened in the second dimension (y-direction) such that theacceleration-based feature data object contains more information sampledover fewer segments in time.

The term “acceleration model FFT layer” may refer to a data object thatdescribes a layer of an acceleration processing machine learning modelconfigured to process an acceleration data object using an FFT functionand generate an acceleration model FFT output. For example, theacceleration model FFT layer may be configured to perform a plurality ofsteps/operations with respect to an acceleration data object describingraw acceleration data in the time to domain to generate an accelerationdata object describing raw acceleration data in the frequency domain.The resulting acceleration model FFT output may have dimensions that aredifferent from the dimensions of the input data object (e.g.,acceleration data object). For example, the acceleration model FFT layermay apply a mask/filter to the input data object (e.g., accelerationdata object) in order to extract relevant features and generate anacceleration model FFT output describing such features.

The term “acceleration model FFT output” may refer to a data object thatdescribes the output generated by an acceleration model FFT layer of anacceleration processing machine learning model. The acceleration modelFFT output may refer to a data object describing a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a plurality of segments in time and a second dimensioncorresponds with a plurality of relevant features.

The term “acceleration model one-dimensional convolutional layer” mayrefer to a data object that describes a layer of a machine learningmodel configured to perform one or more convolutional steps/operationswith respect to an input data object (e.g., an acceleration model FFToutput) and generate an acceleration model convolutional output. Theacceleration model one-dimensional convolutional layer is configured toextract feature data from an acceleration model FFT output. For example,the acceleration model one-dimensional convolutional layer may extractfeature data from an acceleration model FFT output in order to generatean acceleration model convolutional output with different dimensionsfrom the acceleration model FFT output.

The term “acceleration model convolutional output” may refer to a dataobject that describes the output generated by an acceleration modelone-dimensional convolutional layer of an acceleration processingmachine learning model. In some embodiments, the acceleration modelone-dimensional convolutional layer of an acceleration processingmachine learning model processes the acceleration model FFT output of anacceleration model FFT layer in order to generate the acceleration modelconvolutional output.

The term “acceleration model up-sampling layer” may refer to a dataobject that is configured to describe layer of a machine learning modelconfigured to perform one or more up-sampling steps/operations withrespect to an input data object (e.g., acceleration model FFT output oracceleration model convolutional output) in order to generate anacceleration model up-sampling output. The acceleration modelup-sampling layer may be configured to transform (e.g., lengthen ortruncate) at least one dimension of the input data object (e.g.,acceleration model FFT output or acceleration model convolutionaloutput). For example, by increasing the length of at least one dimension(e.g., number of time segments) corresponding with the input dataobject. The acceleration processing machine learning model may comprisea plurality of acceleration model up-sampling layers, each configured totransform a particular dimension of the input data object. Anacceleration model up-sampling output may refer to a data objectdescribing the output generated by the acceleration model up-samplinglayer of an acceleration processing machine learning model.

The term “feature synthesis machine learning model” may refer to a dataobject that is configured to describe parameters and/or hyper-parametersof a machine learning model configured to process two or more input dataobjects (e.g., an audio-based feature data object and anacceleration-based feature data object) in order to generate an inputthat integrates features described by both of the two or more input dataobjects. For example, the feature synthesis machine learning model maybe configured to perform a plurality of steps/operations with respect toan audio-based feature data object and an acceleration-based featureobject in order to generate a hybrid-input prediction data object. Thefeature synthesis machine learning model may comprise one or more layersconfigured to extract sequential information/understanding (e.g.,recognize patterns) in historical data contained in the precedingmachine learning model(s) and/or layers.

The term “acceleration model sequence modeling layer” may refer to adata object that describes a layer of an acceleration model dimensionadjustment layer of an acceleration model dimension adjustment layermachine learning model configured to process an acceleration modelup-sampling output to generate an acceleration model sequence modelingoutput. An acceleration model sequence modeling output may refer to adata object that describes the output of the acceleration model sequencemodeling layer.

The term “hybrid-input prediction data object” may refer to a dataobject that describes the output of two or more input data objects(e.g., an audio-based feature data object and an acceleration-basedfeature data object) generated (e.g., combined) by a feature synthesismachine learning model. An example hybrid-input prediction data objectmay describe a likelihood of an event across one or more time intervals,a frequency of an event across one or more time intervals or a durationof an event across one or more time intervals. For example, ahybrid-input prediction data object may describe a cough likelihood, acough frequency and/or a cough duration for a monitored individualacross one or more time intervals.

The term “target dimension” may refer to a desired length of a dimensionof a data object, such as an acceleration-based feature data objectand/or an audio-based feature data object. In order to performparticular steps/operations (e.g., concatenate) on two or more dataobjects (e.g., matrices), at least one dimension of each data objectmust be equal in length. For instance, in order to perform concatenationand/or addition steps/operations on two data objects of differentdimensions, at least one dimension of the first data object must beadjusted to match the corresponding dimension of the second data object.An audio feature object target dimension length may refer to a desireddimension length for one of the dimensions of an audio-based featuredata object. An acceleration feature object target dimension length mayrefer to a desired dimension length for one of the dimensions of anacceleration-based feature data object.

The term “acceleration model dimension adjustment layer” may refer to adata object that describes a layer of an acceleration processing machinelearning model that is configured to perform a plurality ofsteps/operations with respect to an input data object (e.g., anacceleration model up-sampling output), where the plurality ofsteps/operations are configured to adjust the dimension of anintermediate output of the acceleration processing machine learning inaccordance with a target dimension for the intermediate output. Anexample acceleration model dimension adjustment layer may adjust adimension of the acceleration model up-sampling output to satisfy atarget dimension criteria in accordance with an acceleration featureobject target dimension. The acceleration model dimension adjustmentlayer may comprise one or more layers configured to extract sequentialinformation/understanding (e.g., recognize patterns) in historical datacontained in the preceding machine learning model(s) and/or layers. Theacceleration model dimension adjustment layer may be configured tomodify the sampling rate applied to an input dataset such that one ormore dimensions of the corresponding output are lengthened or truncatedas desired to the target dimension. An acceleration model dimensionadjustment output may refer to a data object that describes the outputof an acceleration model dimension adjustment layer of an accelerationprocessing machine learning model.

The term “model up-sampling layer” may refer to a data object thatdescribes a layer of the acceleration dimension adjustment layer that isconfigured to perform an up-sampling step/operation on an accelerationmodel sequence modeling output to generate a second acceleration modelup-sampling output. An up-sampling output may refer to a data objectthat describes the output of a model up-sampling layer. An accelerationmodel dimension truncation layer may refer to a layer of the dimensionadjustment layer that is configured to process the second accelerationmodel up-sampling output to generate the acceleration model dimensionadjustment output.

The term “concatenation layer” may refer to a data object that describesa layer of the feature synthesis machine learning model that isconfigured to combine (e.g., merge, concatenate, stack and/or the like)two or more data objects (e.g., matrices) to generate an outputdescribing the data in the two or more data objects, where thecombination of the two or more data objects is intended to concentratethe values of the two or more data objects along a concatenationdimension (e.g., a horizontal dimension or a vertical dimension). Forexample, the concatenation layer may be configured to process theaudio-based feature data object and the acceleration-based feature dataobject in order to generate a concatenated feature data object.Concatenation of the audio-based feature data object and theacceleration-based feature data object may include vector-by-vectorsteps/operations to extract new features from the feature data objects.A concatenated feature data object may refer to a data object thatdescribes the output of the concatenation layer.

The term “synthesis model sequence modeling layer” may refer to a dataobject that describes a layer of a feature synthesis machine learningmodel that is configured to perform one or more synthesis model sequencesteps/operations with respect to an input data object (e.g., aconcatenation feature data object) and generate a synthesis modelsequence modeling output. For example, the synthesis model sequencemodeling layer, may further modify (e.g., flatten, combine) an inputdata object to generate the synthesis model sequence modeling output.The synthesis model sequence modeling layer may comprise one or moreGated Recurrent Units (GRUs), Recurrent Neural Networks (RNNs), LongShort-Term Memory (LSTM) units and/or the like.

The term “synthesis model sequence modeling output” may refer to a dataobject that describes the output of one or more synthesis model sequencemodeling layers of a feature synthesis machine learning model. In someembodiments, a feature synthesis machine learning model comprising oneor more synthesis model sequence modeling layer processes aconcatenation layer output of a concatenation layer to generate asynthesis model sequence modeling output.

The term “time-distributed fully connected layer” may refer to a dataobject that describes a fully-connected layer of a feature synthesismachine learning model that is configured to perform one or moreprediction-based actions with respect to an input data object (e.g. asynthesis model sequence modeling output) and generate a hybrid-inputprediction data object. An example of a time-distributed fully connectedlayer is the time-distributed Dense layer in the Keras applicationprogramming interface (API). The time-distributed fully connected layermay process an input data object (e.g., the synthesis model sequencemodeling output) by extracting the most relevant feature for each timesegment to generate a hybrid input prediction data object. Thetime-distributed fully connected layer may be configured to determinethe likelihood of one or more events based at least in part on thehybrid-input prediction data object. The time-distributed fullyconnected layer may be configured to determine a frequency or durationcorresponding with the one or more events across one or more timeintervals. The hybrid input prediction data object may refer to a dataobject that describes the output of the time-distributed fully connectedlayer of the feature synthesis machine learning model. An example hybridinput prediction data object may comprise a single vector comprising asingle relevant feature corresponding with each time segment.

III. Computer Program Products, Methods, and Computing Entities

Embodiments of the present invention may be implemented in various ways,including as computer program products that comprise articles ofmanufacture. Such computer program products may include one or moresoftware components including, for example, software objects, methods,data structures, or the like. A software component may be coded in anyof a variety of programming languages. An illustrative programminglanguage may be a lower-level programming language such as an assemblylanguage associated with a particular hardware framework and/oroperating system platform. A software component comprising assemblylanguage instructions may require conversion into executable machinecode by an assembler prior to execution by the hardware framework and/orplatform. Another example programming language may be a higher-levelprogramming language that may be portable across multiple frameworks. Asoftware component comprising higher-level programming languageinstructions may require conversion to an intermediate representation byan interpreter or a compiler prior to execution.

Other examples of programming languages include, but are not limited to,a macro language, a shell or command language, a job control language, ascript language, a database query or search language, and/or a reportwriting language. In one or more example embodiments, a softwarecomponent comprising instructions in one of the foregoing examples ofprogramming languages may be executed directly by an operating system orother software component without having to be first transformed intoanother form. A software component may be stored as a file or other datastorage construct. Software components of a similar type or functionallyrelated may be stored together such as, for example, in a particulardirectory, folder, or library. Software components may be static (e.g.,pre-established or fixed) or dynamic (e.g., created or modified at thetime of execution).

A computer program product may include non-transitory computer-readablestorage medium storing applications, programs, program modules, scripts,source code, program code, object code, byte code, compiled code,interpreted code, machine code, executable instructions, and/or the like(also referred to herein as executable instructions, instructions forexecution, computer program products, program code, and/or similar termsused herein interchangeably). Such non-transitory computer-readablestorage media include all computer-readable media (including volatileand non-volatile media).

In one embodiment, a non-volatile computer-readable storage medium mayinclude a floppy disk, flexible disk, hard disk, solid-state storage(SSS) (e.g., a solid state drive (SSD), solid state card (SSC), solidstate module (SSM), enterprise flash drive, magnetic tape, or any othernon-transitory magnetic medium, and/or the like. A non-volatilecomputer-readable storage medium may also include a punch card, papertape, optical mark sheet (or any other physical medium with patterns ofholes or other optically recognizable indicia), compact disc read onlymemory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc(DVD), Blu-ray disc (BD), any other non-transitory optical medium,and/or the like. Such a non-volatile computer-readable storage mediummay also include read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), flash memory (e.g.,Serial, NAND, NOR, and/or the like), multimedia memory cards (MMC),secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF)cards, Memory Sticks, and/or the like. Further, a non-volatilecomputer-readable storage medium may also include conductive-bridgingrandom access memory (CBRAM), phase-change random access memory (PRAM),ferroelectric random-access memory (FeRAM), non-volatile random-accessmemory (NVRAM), magnetoresistive random-access memory (MRAM), resistiverandom-access memory (RRAM), Silicon-Oxide-Nitride-Oxide-Silicon memory(SONOS), floating junction gate random access memory (FJG RAM),Millipede memory, racetrack memory, and/or the like.

In one embodiment, a volatile computer-readable storage medium mayinclude random access memory (RAM), dynamic random access memory (DRAM),static random access memory (SRAM), fast page mode dynamic random accessmemory (FPM DRAM), extended data-out dynamic random access memory (EDODRAM), synchronous dynamic random access memory (SDRAM), double datarate synchronous dynamic random access memory (DDR SDRAM), double datarate type two synchronous dynamic random access memory (DDR2 SDRAM),double data rate type three synchronous dynamic random access memory(DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), TwinTransistor RAM (TTRAM), Thyristor RAM (T-RAM), Zero-capacitor (Z-RAM),Rambus in-line memory module (RIMM), dual in-line memory module (DIMM),single in-line memory module (SIMM), video random access memory (VRAM),cache memory (including various levels), flash memory, register memory,and/or the like. It will be appreciated that where embodiments aredescribed to use a computer-readable storage medium, other types ofcomputer-readable storage media may be substituted for or used inaddition to the computer-readable storage media described above.

As should be appreciated, various embodiments of the present inventionmay also be implemented as methods, apparatuses, systems, computingdevices, computing entities, and/or the like. As such, embodiments ofthe present invention may take the form of an apparatus, system,computing device, computing entity, and/or the like executinginstructions stored on a computer-readable storage medium to performcertain steps or operations. Thus, embodiments of the present inventionmay also take the form of an entirely hardware embodiment, an entirelycomputer program product embodiment, and/or an embodiment that comprisescombination of computer program products and hardware performing certainsteps or operations.

Embodiments of the present invention are described below with referenceto block diagrams and flowchart illustrations. Thus, it should beunderstood that each block of the block diagrams and flowchartillustrations may be implemented in the form of a computer programproduct, an entirely hardware embodiment, a combination of hardware andcomputer program products, and/or apparatuses, systems, computingdevices, computing entities, and/or the like carrying out instructions,steps or operations, and similar words used interchangeably (e.g., theexecutable instructions, instructions for execution, program code,and/or the like) on a computer-readable storage medium for execution.For example, retrieval, loading, and execution of code may be performedsequentially such that one instruction is retrieved, loaded, andexecuted at a time. In some exemplary embodiments, retrieval, loading,and/or execution may be performed in parallel such that multipleinstructions are retrieved, loaded, and/or executed together. Thus, suchembodiments can produce specifically-configured machines performing thesteps or operations specified in the block diagrams and flowchartillustrations. Accordingly, the block diagrams and flowchartillustrations support various combinations of embodiments for performingthe specified instructions, steps or operations.

IV. Exemplary System Framework

FIG. 1 is a schematic diagram of an example architecture 100 forperforming hybrid-input prediction steps/operations. The architecture100 includes a hybrid-input prediction system 101 configured to receivedata from the client computing entities 102, process the data togenerate outputs (e.g., hybrid-input predictive data objects) andprovide the outputs to the client computing entities 102 for generatingcorresponding alerts (e.g., for providing and/or updating a userinterface). In some embodiments, hybrid-input prediction system 101 maycommunicate with at least one of the client computing entities 102 usingone or more communication networks. Examples of communication networksinclude any wired or wireless communication network including, forexample, a wired or wireless local area network (LAN), personal areanetwork (PAN), metropolitan area network (MAN), wide area network (WAN),or the like, as well as any hardware, software and/or firmware requiredto implement it (such as, e.g., network routers, and/or the like).

The hybrid-input prediction system 101 may include a hybrid-inputpredictive computing entity 106, and a storage subsystem 108. Thehybrid-input predictive computing entity 106 may be configured toreceive requests and/or data from client computing entities 102, processthe requests and/or data to generate predictive outputs (e.g.,hybrid-input prediction data objects), and provide the predictiveoutputs to the client computing entities 102. The client computingentities 102 may be triggered to transmit requests to the hybrid-inputpredictive computing entity 106 in response to events which satisfycertain parameters (e.g., monitored events). Responsive to receiving thepredictive outputs, the client computing entities 102 may generatecorresponding alerts and may provide (e.g., transmit, send and/or thelike) corresponding user interface data for presentation to usercomputing entities.

The storage subsystem 108 may be configured to store at least a portionof the data utilized by the hybrid-input predictive computing entity 106to perform hybrid-input prediction steps/operations and tasks. Thestorage subsystem 108 may be configured to store at least a portion ofoperational data and/or operational configuration data includingoperational instructions and parameters utilized by the hybrid-inputpredictive computing entity 106 to perform hybrid-input predictionsteps/operations in response to requests. The storage subsystem 108 mayinclude one or more storage units, such as multiple distributed storageunits that are connected through a computer network. Each storage unitin the storage subsystem 108 may store at least one of one or more dataassets and/or one or more data about the computed properties of one ormore data assets. Moreover, each storage unit in the storage subsystem108 may include one or more non-volatile storage or memory mediaincluding but not limited to hard disks, ROM, PROM, EPROM, EEPROM, flashmemory, MMCs, SD memory cards, Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM,MRAM, RRAM, SONOS, FJG RAM, Millipede memory, racetrack memory, and/orthe like.

Exemplary Hybrid-Input Predictive Computing Entity

FIG. 2 provides a schematic of a hybrid-input predictive computingentity 106 according to one embodiment of the present invention. Ingeneral, the terms computing entity, computer, entity, device, system,and/or similar words used herein interchangeably may refer to, forexample, one or more computers, computing entities, desktops, mobilephones, tablets, phablets, notebooks, laptops, distributed systems,kiosks, input terminals, servers or server networks, blades, gateways,switches, processing devices, processing entities, set-top boxes,relays, routers, network access points, base stations, the like, and/orany combination of devices or entities adapted to perform the functions,steps/operations, and/or processes described herein. Such functions,steps/operations, and/or processes may include, for example,transmitting, receiving, operating on, processing, displaying, storing,determining, creating/generating, monitoring, evaluating, comparing,and/or similar terms used herein interchangeably. In one embodiment,these functions, steps/operations, and/or processes can be performed ondata, content, information, and/or similar terms used hereininterchangeably.

As indicated, in one embodiment, the hybrid-input predictive computingentity 106 may also include one or more network interfaces 220 forcommunicating with various computing entities, such as by communicatingdata, content, information, and/or similar terms used hereininterchangeably that can be transmitted, received, operated on,processed, displayed, stored, and/or the like.

As shown in FIG. 2 , in one embodiment, the hybrid-input predictivecomputing entity 106 may include or be in communication with one or moreprocessing elements 205 (also referred to as processors, processingcircuitry, and/or similar terms used herein interchangeably) thatcommunicate with other elements within the hybrid-input predictivecomputing entity 106 via a bus, for example. As will be understood, theprocessing element 205 may be embodied in a number of different ways.

For example, the processing element 205 may be embodied as one or morecomplex programmable logic devices (CPLDs), microprocessors, multi-coreprocessors, coprocessing entities, application-specific instruction-setprocessors (ASIPs), microcontrollers, and/or controllers. Further, theprocessing element 205 may be embodied as one or more other processingdevices or circuitry. The term circuitry may refer to an entirelyhardware embodiment or a combination of hardware and computer programproducts. Thus, the processing element 205 may be embodied as integratedcircuits, application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), programmable logic arrays (PLAs),hardware accelerators, other circuitry, and/or the like.

As will therefore be understood, the processing element 205 may beconfigured for a particular use or configured to execute instructionsstored in volatile or non-volatile media or otherwise accessible to theprocessing element 205. As such, whether configured by hardware orcomputer program products, or by a combination thereof, the processingelement 205 may be capable of performing steps or operations accordingto embodiments of the present invention when configured accordingly.

In one embodiment, the hybrid-input predictive computing entity 106 mayfurther include or be in communication with non-volatile media (alsoreferred to as non-volatile storage, memory, memory storage, memorycircuitry and/or similar terms used herein interchangeably). In oneembodiment, the non-volatile storage or memory may include one or morenon-volatile storage or memory media 210, including but not limited tohard disks, ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memorycards, Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJGRAM, Millipede memory, racetrack memory, and/or the like.

As will be recognized, the non-volatile storage or memory media maystore databases, database instances, database management systems, data,applications, programs, program modules, scripts, source code, objectcode, byte code, compiled code, interpreted code, machine code,executable instructions, and/or the like. The term database, databaseinstance, database management system, and/or similar terms used hereininterchangeably may refer to a collection of records or data that isstored in a computer-readable storage medium using one or more databasemodels, such as a hierarchical database model, network model, relationalmodel, entity—relationship model, object model, document model, semanticmodel, graph model, and/or the like.

In one embodiment, the hybrid-input predictive computing entity 106 mayfurther include or be in communication with volatile media (alsoreferred to as volatile storage, memory, memory storage, memorycircuitry and/or similar terms used herein interchangeably). In oneembodiment, the volatile storage or memory may also include one or morevolatile storage or memory media 215, including but not limited to RAM,DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory,register memory, and/or the like.

As will be recognized, the volatile storage or memory media may be usedto store at least portions of the databases, database instances,database management systems, data, applications, programs, programmodules, scripts, source code, object code, byte code, compiled code,interpreted code, machine code, executable instructions, and/or the likebeing executed by, for example, the processing element 205. Thus, thedatabases, database instances, database management systems, data,applications, programs, program modules, scripts, source code, objectcode, byte code, compiled code, interpreted code, machine code,executable instructions, and/or the like may be used to control certainaspects of the operation of the hybrid-input predictive computing entity106 with the assistance of the processing element 205 and operatingsystem.

As indicated, in one embodiment, the hybrid-input predictive computingentity 106 may also include one or more network interfaces 220 forcommunicating with various computing entities, such as by communicatingdata, content, information, and/or similar terms used hereininterchangeably that can be transmitted, received, operated on,processed, displayed, stored, and/or the like. Such communication may beexecuted using a wired data transmission protocol, such as fiberdistributed data interface (FDDI), digital subscriber line (DSL),Ethernet, asynchronous transfer mode (ATM), frame relay, data over cableservice interface specification (DOCSIS), or any other wiredtransmission protocol. Similarly, the hybrid-input predictive computingentity 106 may be configured to communicate via wireless clientcommunication networks using any of a variety of protocols, such asgeneral packet radio service (GPRS), Universal Mobile TelecommunicationsSystem (UMTS), Code Division Multiple Access 2000 (CDMA2000), CDMA20001× (1×RTT), Wideband Code Division Multiple Access (WCDMA), GlobalSystem for Mobile Communications (GSM), Enhanced Data rates for GSMEvolution (EDGE), Time Division-Synchronous Code Division MultipleAccess (TD-SCDMA), Long Term Evolution (LTE), Evolved UniversalTerrestrial Radio Access Network (E-UTRAN), Evolution-Data Optimized(EVDO), High Speed Packet Access (HSPA), High-Speed Downlink PacketAccess (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX),ultra-wideband (UWB), infrared (IR) protocols, near field communication(NFC) protocols, Wibree, Bluetooth protocols, wireless universal serialbus (USB) protocols, and/or any other wireless protocol.

Although not shown, the hybrid-input predictive computing entity 106 mayinclude or be in communication with one or more input elements, such asa keyboard input, a mouse input, a touch screen/display input, motioninput, movement input, audio input, pointing device input, joystickinput, keypad input, and/or the like. The hybrid-input predictivecomputing entity 106 may also include or be in communication with one ormore output elements (not shown), such as audio output, video output,screen/display output, motion output, movement output, and/or the like.

Exemplary Client Computing Entity

FIG. 3 provides an illustrative schematic representative of a clientcomputing entity 102 that can be used in conjunction with embodiments ofthe present invention. In general, the terms device, system, computingentity, entity, and/or similar words used herein interchangeably mayrefer to, for example, one or more computers, computing entities,desktops, mobile phones, tablets, phablets, notebooks, laptops,distributed systems, kiosks, input terminals, servers or servernetworks, blades, gateways, switches, processing devices, processingentities, set-top boxes, relays, routers, network access points, basestations, the like, and/or any combination of devices or entitiesadapted to perform the functions, steps/operations, and/or processesdescribed herein. Client computing entities 102 can be operated byvarious parties. As shown in FIG. 3 , the client computing entity 102can include an antenna 312, a transmitter 304 (e.g., radio), a receiver306 (e.g., radio), and a processing element 308 (e.g., CPLDs,microprocessors, multi-core processors, coprocessing entities, ASIPs,microcontrollers, and/or controllers) that provides signals to andreceives signals from the transmitter 304 and receiver 306,correspondingly.

The signals provided to and received from the transmitter 304 and thereceiver 306, correspondingly, may include signaling information/data inaccordance with air interface standards of applicable wireless systems.In this regard, the client computing entity 102 may be capable ofoperating with one or more air interface standards, communicationprotocols, modulation types, and access types. More particularly, theclient computing entity 102 may operate in accordance with any of anumber of wireless communication standards and protocols, such as thosedescribed above with regard to the hybrid-input predictive computingentity 106. In a particular embodiment, the client computing entity 102may operate in accordance with multiple wireless communication standardsand protocols, such as UMTS, CDMA2000, 1×RTT, WCDMA, GSM, EDGE,TD-SCDMA, LTE, E-UTRAN, EVDO, HSPA, HSDPA, Wi-Fi, Wi-Fi Direct, WiMAX,UWB, IR, NFC, Bluetooth, USB, and/or the like. Similarly, the clientcomputing entity 102 may operate in accordance with multiple wiredcommunication standards and protocols, such as those described abovewith regard to the hybrid-input predictive computing entity 106 via anetwork interface 320.

Via these communication standards and protocols, the client computingentity 102 can communicate with various other entities using conceptssuch as Unstructured Supplementary Service Data (USSD), Short MessageService (SMS), Multimedia Messaging Service (MMS), Dual-ToneMulti-Frequency Signaling (DTMF), and/or Subscriber Identity ModuleDialer (SIM dialer). The client computing entity 102 can also downloadchanges, add-ons, and updates, for instance, to its firmware, software(e.g., including executable instructions, applications, programmodules), and operating system.

According to one embodiment, the client computing entity 102 may includelocation determining aspects, devices, modules, functionalities, and/orsimilar words used herein interchangeably. For example, the clientcomputing entity 102 may include outdoor positioning aspects, such as alocation module adapted to acquire, for example, latitude, longitude,altitude, geocode, course, direction, heading, speed, universal time(UTC), date, and/or various other information/data. In one embodiment,the location module can acquire data, sometimes known as ephemeris data,by identifying the number of satellites in view and the relativepositions of those satellites (e.g., using global positioning systems(GPS)). The satellites may be a variety of different satellites,including Low Earth Orbit (LEO) satellite systems, Department of Defense(DOD) satellite systems, the European Union Galileo positioning systems,the Chinese Compass navigation systems, Indian Regional Navigationalsatellite systems, and/or the like. This data can be collected using avariety of coordinate systems, such as the Decimal Degrees (DD);Degrees, Minutes, Seconds (DMS); Universal Transverse Mercator (UTM);Universal Polar Stereographic (UPS) coordinate systems; and/or the like.Alternatively, the location information/data can be determined bytriangulating the client computing entity's 102 position in connectionwith a variety of other systems, including cellular towers, Wi-Fi accesspoints, and/or the like. Similarly, the client computing entity 102 mayinclude indoor positioning aspects, such as a location module adapted toacquire, for example, latitude, longitude, altitude, geocode, course,direction, heading, speed, time, date, and/or various otherinformation/data. Some of the indoor systems may use various position orlocation technologies including RFID tags, indoor beacons ortransmitters, Wi-Fi access points, cellular towers, nearby computingdevices (e.g., smartphones, laptops) and/or the like. For instance, suchtechnologies may include the iBeacons, Gimbal proximity beacons,Bluetooth Low Energy (BLE) transmitters, NFC transmitters, and/or thelike. These indoor positioning aspects can be used in a variety ofsettings to determine the location of someone or something to withininches or centimeters.

The client computing entity 102 may also comprise a user interface (thatcan include a display 316 coupled to a processing element 308) and/or auser input interface (coupled to a processing element 308). For example,the user interface may be a user application, browser, user interface,and/or similar words used herein interchangeably executing on and/oraccessible via the client computing entity 102 to interact with and/orcause display of information/data from the hybrid-input predictivecomputing entity 106, as described herein. The user input interface cancomprise any of a number of devices or interfaces allowing the clientcomputing entity 102 to receive data, such as a keypad 318 (hard orsoft), a touch display, voice/speech or motion interfaces, or otherinput device. In embodiments including a keypad 318, the keypad 318 caninclude (or cause display of) the conventional numeric (0-9) and relatedkeys (#, *), and other keys used for operating the client computingentity 102 and may include a full set of alphabetic keys or set of keysthat may be activated to provide a full set of alphanumeric keys. Inaddition to providing input, the user input interface can be used, forexample, to activate or deactivate certain functions, such as screensavers and/or sleep modes.

The client computing entity 102 can also include volatile storage ormemory 322 and/or non-volatile storage or memory 324, which can beembedded and/or may be removable. For example, the non-volatile memorymay be ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards,Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM,Millipede memory, racetrack memory, and/or the like. The volatile memorymay be RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM,cache memory, register memory, and/or the like. The volatile andnon-volatile storage or memory can store databases, database instances,database management systems, data, applications, programs, programmodules, scripts, source code, object code, byte code, compiled code,interpreted code, machine code, executable instructions, and/or the liketo implement the functions of the client computing entity 102. Asindicated, this may include a user application that is resident on theentity or accessible through a browser or other user interface forcommunicating with the hybrid-input predictive computing entity 106and/or various other computing entities.

In another embodiment, the client computing entity 102 may include oneor more components or functionality that are the same or similar tothose of the hybrid-input predictive computing entity 106, as describedin greater detail above. As will be recognized, these frameworks anddescriptions are provided for exemplary purposes only and are notlimiting to the various embodiments.

In various embodiments, the client computing entity 102 may be embodiedas an artificial intelligence (AI) computing entity, such as an AmazonEcho, Amazon Echo Dot, Amazon Show, Google Home, and/or the like.Accordingly, the client computing entity 102 may be configured toprovide and/or receive information/data from a user via an input/outputmechanism, such as a display, a camera, a speaker, a voice-activatedinput, and/or the like. In certain embodiments, an AI computing entitymay comprise one or more predefined and executable program algorithmsstored within an onboard memory storage module, and/or accessible over anetwork. In various embodiments, the AI computing entity may beconfigured to retrieve and/or execute one or more of the predefinedprogram algorithms upon the occurrence of a predefined trigger event.

V. Exemplary System Operations

Described herein are various techniques for hybrid-input predictionsteps/operations on two or more dissimilar data objects. Some of thedisclosed techniques may utilize one or more machine learning models toperform hybrid-input prediction steps/operations (e.g., convolutionalsteps/operations, up-sampling steps/operations and the like). Some ofthe described techniques utilize a particular configuration of machinelearning models and/or layers. The output of a machine learning modeland/or layers therein may be supplied as an input for subsequentsteps/operations by another machine learning model and/or layer.However, a person of ordinary skill in the art will recognize thathybrid-input prediction steps/operations discussed herein may beperformed using different combinations than the particular combinationsdescribed herein.

By facilitating efficient hybrid-input prediction steps/operations,various embodiments of the present invention improve hybrid-inputprediction steps/operations on two or more dissimilar data objects.Modifying data objects according to the methods disclosed herein reducesthe need for additional processing power and in turn reduces processinglatency. This in turn reduces the need for large amounts of data toprovide accurate predictive outputs.

FIG. 4 provides a schematic representation of an example system 400 forperforming hybrid-input prediction steps/operations and generatingcorresponding predictive outputs/user interface data. The hybrid-inputprediction system 101 may receive data from one or more client computingentities 102 and store at least a portion of the data in the storagesubsystem 108. The storage subsystem 108 may provide acceleration andaudio data objects 402, 404 to the hybrid-input predictive computingentity 106. The hybrid-input prediction system 101 may be configured toperform steps/operations that lead to generating hybrid-input predictiondata objects 411 and user interface data. For example, the hybrid-inputpredictive computing entity 106 may comprise a plurality of machinelearning models each configured to perform an associated set ofhybrid-input prediction steps/operations. The hybrid-input predictionsystem 101 may process two or more input data objects (e.g., anacceleration data object 402, and an audio data object 404) and generatea predictive output (e.g., one or more hybrid-input prediction dataobjects 411).

Exemplary Acceleration Processing Machine Learning Model

As illustrated in FIG. 4 , the hybrid-input prediction system 101comprises an acceleration processing machine learning model 401configured to process an acceleration data object 402 and generate anacceleration-based feature data object 406. The acceleration processingmachine learning model 401 may refer to a machine learning modelconfigured to perform a plurality of steps/operations with respect to anacceleration data object 402. An acceleration data object 402 may referto a data object that describes a set of information (e.g., an inputdataset) corresponding to acceleration of motion of a body (e.g., ahuman) with respect to the body's frame (e.g., raw acceleration data).An example acceleration data object 402 may describe a body's change inorientation and/or movements (e.g., vibrations) in two or moredirections (e.g., recorded cross-body acceleration data for a monitoredindividual). Raw acceleration data may be represented graphically as awaveform plotted with respect to time in which positive values indicatean increase in velocity, negative values indicate a decrease in velocityand zero/null values indicate a constant velocity. An acceleration dataobject 402 may describe such raw acceleration data in a time domainrepresentation or the frequency domain representation by sampling theraw acceleration data at a given sampling rate (e.g., 50 Hz). Samplingraw acceleration data at a high sampling rate (e.g., 44.1 kHz) isundesirable due to a lack of complexity in the underlying data. Anacceleration data object 402 describing raw acceleration data in thetime domain may comprise a two-dimensional matrix having a length and awidth in which a first dimension corresponds with a number of segmentsin time and a second dimension corresponds with a plurality of featuresassociated with oscillations at different frequencies occurring atcorresponding segments in time. An acceleration-based feature dataobject 406 may refer to a data object that is generated by processing anacceleration data object 402 using an acceleration processing machinelearning model 401. An example acceleration-based feature data object406 may comprise a two-dimensional matrix having a length and a width inwhich a first dimension corresponds with a number of segments in timeand a second dimension corresponds with a plurality of featuresassociated with oscillations/vibrations occurring at the correspondingsegments in time. The acceleration-based feature data object 406 maycomprise dimensions that are different from the dimensions of the inputdata object (e.g., acceleration data object 402) processed by theacceleration processing machine learning model 401. For instance, thedimensions of the matrix of the acceleration-based feature data object406 may be truncated in the first dimension (x-direction) and lengthenedin the second dimension (y-direction) such that the acceleration-basedfeature data object 406 contains more information sampled over fewersegments in time.

FIG. 10 is a flowchart diagram illustrating an example process forgenerating an acceleration-based feature data object 406 by anacceleration processing machine learning model 401. The process depictedin FIG. 10 begins at step/operation 1002, when the accelerationprocessing machine learning model 401 processes the acceleration dataobject 402 to generate the acceleration model FFT output 512. Atstep/operation 1004, the acceleration processing machine learning model401 processes the acceleration model FFT output 512 to generate anacceleration model convolutional output 514. At step/operation 1006, theacceleration processing machine learning model 401 processes theacceleration model convolutional output 514 to generate an accelerationmodel up-sampling output 516. Then, at step/operation 1008, theacceleration processing machine learning model 401 generates theacceleration-based feature data object 406 based at least in part on theacceleration model up-sampling output 516.

FIG. 5 provides a schematic representation of an example system 500 forgenerating an acceleration-based feature data object 406 by anacceleration processing machine learning model 401. The accelerationprocessing machine learning model 401 may comprise a plurality of layersconfigured to process the acceleration data object 402 and generate anacceleration-based feature data object 406.

As illustrated in FIG. 5 the acceleration processing machine learningmodel 401 comprises an acceleration model FFT layer 501 configured toprocess an acceleration data object 402 using an FFT function andgenerate an acceleration model FFT output 512. The acceleration modelFFT layer 501 may refer to a layer of an acceleration processing machinelearning model 401 configured to process an acceleration data object 402using an FFT function and generate an acceleration model FFT output 512.For example, the acceleration model FFT layer 501 may be configured toperform a plurality of steps/operations with respect to an accelerationdata object 402 describing raw acceleration data in the time to domainto generate an acceleration data object 402 describing raw accelerationdata in the frequency domain. The resulting acceleration model FFToutput 512 may have dimensions that are different from the dimensions ofthe input data object (e.g., acceleration data object 402). For example,the acceleration model FFT layer 501 may apply a mask/filter to theinput data object (e.g., acceleration data object 402) in order toextract relevant features and generate the acceleration model FFT output512 describing such features. The acceleration model FFT output 512 mayrefer to the output generated by an acceleration model FFT layer 501 ofan acceleration processing machine learning model 401. The accelerationmodel FFT output 512 may refer to a data object describing atwo-dimensional matrix having a length and a width in which a firstdimension corresponds with a plurality of segments in time and a seconddimension corresponds with a plurality of relevant features.

As illustrated in FIG. 5 , the acceleration processing machine learningmodel 401 comprises an acceleration model one-dimensional convolutionallayer 503 configured to perform one or more convolutionalsteps/operations with respect to an acceleration model FFT output 512and generate an acceleration model convolutional output 514. Anacceleration model one-dimensional convolutional layer 503 may refer toa layer of a machine learning model configured to perform one or moreconvolutional steps/operations with respect to an input data object(e.g., an acceleration model FFT output 512) and generate anacceleration model convolutional output 514. The acceleration modelone-dimensional convolutional layer 503 is configured to extract featuredata from an acceleration model FFT output 512. For example, theacceleration model one-dimensional convolutional layer 503 may extractfeature data from an acceleration model FFT output 512 in order togenerate an acceleration model convolutional output 514 with differentdimensions from the acceleration model FFT output 512. An accelerationmodel convolutional output 514 may refer to a data object that describesthe output generated by an acceleration model one-dimensionalconvolutional layer 503 of an acceleration processing machine learningmodel 401.

As illustrated in FIG. 5 , the acceleration processing machine learningmodel 401 comprises an acceleration model up-sampling layer 505configured to perform one or more up-sampling steps/operations withrespect to an acceleration model convolutional output 514 and generatean acceleration model up-sampling output 516. The acceleration modelup-sampling layer 505 may refer to a layer of an acceleration processingmachine learning model 401 configured to perform one or more up-samplingsteps/operations with respect to an input data object (e.g.,acceleration model convolutional output 514) in order to generate anacceleration model up-sampling output 516. The acceleration modelup-sampling layer 505 may be configured to transform (e.g., lengthen ortruncate) at least one dimension of the input data object (e.g.,acceleration model convolutional output 514). For example, by increasingthe length of at least one dimension (e.g., number of time segments)corresponding with the input data object. The acceleration processingmachine learning model 401 may comprise a plurality of accelerationmodel up-sampling layers 505, each configured to transform a particulardimension of a respective input data object. An acceleration modelup-sampling output 516 may refer to a data object describing the outputgenerated by the acceleration model up-sampling layer 505 of anacceleration processing machine learning model 401. As furtherillustrated in FIG. 5 , the acceleration processing machine learningmodel 401 comprises an acceleration model dimension adjustment layer 507configured to perform one or more dimension adjustment steps/operationswith respect to an acceleration model up-sampling output 516 andgenerate an acceleration model dimension adjustment output 518.

FIG. 8 provides a schematic 800 illustrating an exemplary accelerationmodel dimension adjustment layer 507. The acceleration model dimensionadjustment layer 507 may comprise one or more layers of the accelerationprocessing machine learning model 401 configured to process theacceleration model up-sampling output 516 and generate the accelerationmodel dimension adjustment output 518. The acceleration model dimensionadjustment layer 507 may refer to one or more layers of an accelerationprocessing machine learning model 401 configured to perform a pluralityof up-sampling steps/operations with respect to an input data object(e.g., an acceleration model up-sampling output 516). The plurality ofsteps/operations are configured to adjust the dimension of anintermediate output of the acceleration processing machine learningmodel 401 in accordance with a target dimension for the intermediateoutput. An example acceleration model dimension adjustment layer 507 mayadjust a dimension of the acceleration model up-sampling output tosatisfy target dimension criteria in accordance with anacceleration-based feature object target dimension. The target dimensionmay refer to a desired length of a dimension of a data object. In orderto perform particular steps/operations (e.g., concatenate) on two ormore data objects (e.g., matrices), at least one dimension of each dataobject must be equal in length. For instance, in order to performsteps/operations on two data objects of different dimensions, at leastone dimension of the first data object must be adjusted to match thecorresponding dimension of the second data object. An audio featureobject target dimension length may refer to a desired dimension lengthfor one of the dimensions of an audio-based feature data object. Anacceleration feature object target dimension length may refer to adesired dimension length for one of the dimensions of anacceleration-based feature data object. The acceleration model dimensionadjustment layer 507 may comprise one or more layers configured toextract sequential information/understanding (e.g., recognize patterns)in historical data contained in the preceding machine learning model(s)and/or layers. The acceleration model dimension adjustment layer 507 maybe configured to modify the sampling rate applied to an input datasetsuch that one or more dimensions of the corresponding output arelengthened or truncated as desired to the target dimension. Anacceleration model dimension adjustment output 518 may refer to a dataobject that describes the output of an acceleration model dimensionadjustment layer 507 of an acceleration processing machine learningmodel 401.

As illustrated in FIG. 8 , the acceleration model dimension adjustmentlayer 507 comprises an acceleration model sequence modeling layer 801configured to process an acceleration model up-sampling output 516 togenerate an acceleration model sequence modeling output 812. Theacceleration model sequence modeling layer 801 may refer to a layer ofan acceleration model dimension adjustment layer 507 of an accelerationprocessing machine learning model 401 configured to process anacceleration model up-sampling output 516 to generate an accelerationmodel sequence modeling output. An acceleration model sequence modelingoutput 812 may refer to a data object that describes the output of theacceleration model sequence modeling layer 801. As further illustratedin FIG. 8 , the acceleration model dimension adjustment layer 507comprises a second acceleration up-sampling layer 803 configured toprocess the acceleration model sequence modeling output 812 to generatea second acceleration model up-sampling output 814. While variousembodiments of the present invention disclose using two up-samplinglayers by an acceleration model dimension adjustment layer, a person ofordinary skill in the relevant technology will recognize that anacceleration model dimension adjustment layer may comprise any number ofup-sampling layers. As further illustrated in FIG. 8 , the accelerationmodel dimension adjustment layer 507 comprises an acceleration modeldimension truncation layer 805 configured to process the secondacceleration model up-sampling output 814 to generate the accelerationmodel dimension adjustment output 518.

FIG. 12 is a flowchart diagram illustrating an example process forgenerating the acceleration model dimension adjustment output 518 by theacceleration model dimension adjustment layer 507. The process depictedin FIG. 12 begins at step/operation 1202, when the acceleration modelsequence modeling layer 801 of the acceleration model dimensionadjustment layer 507 processes the acceleration model up-sampling outputby adjusting a target dimension in accordance with the accelerationfeature object target dimension to generate the acceleration modelsequence modeling output 812. At step/operation 1204, the secondacceleration up-sampling layer 803 of the acceleration model dimensionadjustment layer 507 processes the acceleration modeling sequencemodeling output to generate a second acceleration model up-samplingoutput 814. At step/operation 1206, the acceleration model dimensiontruncation layer 805 or the acceleration model dimension adjustmentlayer 507 processes the second acceleration model up-sampling layer togenerate the acceleration model dimension adjustment output 518.

Exemplary Audio Processing Machine Learning Model

Returning to FIG. 4 , the hybrid-input prediction system 101 comprisesan audio processing machine learning model 403 configured to process anaudio data object 404 and generate an audio-based feature data object408. The audio processing machine learning model 403 may refer to amachine learning model configured to perform a plurality ofsteps/operations with respect to an audio data object 404. An audio dataobject 404 may refer to may refer to a data object that describes a setof information (e.g., an input dataset) corresponding to sound such asraw audio data (e.g., recorded audio data for a monitored individual).Raw audio data may comprise one or more sound waves. Each sound wavecomprises a wavelength oscillating at a given frequency for a durationof time. An audio data object 404 may describe such raw audio data in atime domain representation or the frequency domain representation. Rawaudio data may be sampled at a given sampling rate (e.g., 44.1 kHz) togenerate a time domain representation. In general, a data objectgenerated based at least in part on a high sampling rate will generate adata object with more information/data than a data object generatedbased at least in part on a lower sampling rate. An example audio dataobject 404 may be represented graphically by plotting extractedvalues/features as a function of time. An audio data object 404describing raw audio data in the time domain may comprise atwo-dimensional matrix in which a first dimension corresponds with anumber of segments in time and a second dimension corresponds with aplurality of values/features based at least in part on the frequenciesof the sound waves occurring at corresponding segments in time. Anaudio-based feature data object 408 may refer to a data object that isgenerated by an audio processing machine learning model 403. An exampleaudio-based feature data object 408 may comprise a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a number of segments in time and a second dimensioncorresponds with a plurality of features associated with frequencies ofsound waves occurring at the corresponding segments in time. Theaudio-based feature data object 408 may comprise dimensions that aredifferent from the dimensions of the input data object (e.g., audio dataobject 404) processed by the audio processing machine learning model403. For instance, the dimensions of the matrix of the audio-basedfeature data object 408 may be truncated in the first dimension(x-direction) and lengthened in the second dimension (y-direction) suchthat the audio-based feature data object 408 contains more informationsampled over fewer segments in time.

FIG. 6 a schematic representation of an example system 600 forgenerating an audio-based feature data object 408 by an audio processingmachine learning model 403. The audio processing machine learning model403 may comprise a plurality of layers configured to process the audiodata object 404 and generate an audio-based feature data object 408. Theaudio processing machine learning model 403 may refer to a machinelearning model configured to perform a plurality of steps/operationswith respect to an audio data object 404 in order to generate anaudio-based feature data object 408. For example, the audio processingmachine learning model 403 may comprise a plurality of layers eachconfigured to perform one or more steps/operations with respect to aninput data object (e.g., audio data object 404). Each layer of the audioprocessing machine learning model 403 may be configured to perform aplurality of steps/operations to modify one or more dimensions of acorresponding input (e.g., audio data object 404). An example audioprocessing machine learning model 403 may comprise at least one audiomodel fast Fourier transform layer 601, and at least one audio modelone-dimensional convolutional layer 603.

As illustrated in FIG. 6 , the audio processing machine learning model403 comprises an audio model fast Fourier transform (FFT) layer 601configured to process an audio data object 404 using an FFT function andgenerate an audio model FFT output 612. The audio model FFT layer 601 isconfigured to perform a plurality of steps/operations with respect to anaudio data object 404 describing raw audio data in the time domain inorder to generate an audio data object describing raw audio data in thefrequency domain. The resulting audio model FFT output 612 may havedimensions that are different from the dimensions of the input dataobject (e.g., audio data object 404). For example, the audio model FFTlayer 601 may apply a mask/filter to the input data object (e.g., audiodata object 404) in order to extract relevant features and generate theaudio model FFT output 612 describing such features. The audio model FFToutput 612 may refer to the output generated by an audio model FFT layer601 of an audio processing machine learning model 403. The audio modelFFT output 612 may refer to a data object describing a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a plurality of segments in time and a second dimensioncorresponds with a plurality of relevant features.

As illustrated in FIG. 6 , the audio processing machine learning model403 comprises an audio model one-dimensional convolutional layer 603configured to perform one or more convolutional steps/operations withrespect to an audio model FFT output 612 and generate an audio modelconvolutional output 614. The audio model one-dimensional convolutionallayer 603 is configured to perform one or more convolutionalsteps/operations with respect to an audio model FFT output 612 andgenerate an audio model convolutional output 614. The audio modelone-dimensional convolutional layer 603 is configured to extract featuredata from the audio model FFT output 612. For example, the audio modelone-dimensional convolutional layer 603 may extract feature data fromthe audio model FFT output 612 in order to generate an audio-basedfeature data object 408 with different dimensions from the audio modelFFT output 612. An audio model convolutional output 614 may refer to theoutput or intermediary output generated by an audio modelone-dimensional convolutional layer 603 of an audio processing machinelearning model 403.

FIG. 9 is a flowchart diagram illustrating an example process forgenerating an audio-based feature data object 408 by an audio processingmachine learning model 403. The process depicted in FIG. 9 begins atstep/operation 902, when the audio model FFT layer 601 of the audioprocessing machine learning model 403 processes the audio data object404 to generate an audio model FFT output 612. At step/operation 904,the audio model one-dimensional convolutional layer 603 or the audioprocessing machine learning model 403 processes the audio model FFToutput 612 to generate an audio model convolutional output 614. Atstep/operation 906, the audio processing machine learning modelgenerates an audio-based feature data object 408 based at least in parton the audio model convolutional output 614.

Exemplary Feature Synthesis Machine Learning Model

As further illustrated in FIG. 4 , the hybrid-input prediction system101 comprises a feature synthesis machine learning model 405. Thefeature synthesis machine learning model 405 may receive, as input, theacceleration-based feature data object 406 and the audio-based featuredata object 408. The feature synthesis machine learning model 405 mayprocess the acceleration-based feature data object 406 and theaudio-based feature data object 408 and generate a hybrid-inputprediction data object 411. The feature synthesis machine learning model405 may refer to a machine learning model configured to process two ormore input data objects (e.g., an audio-based feature data object 408and an acceleration-based feature data object 406) in order to generatean input that integrates features described by both of the two or moreinput data objects. For example, the feature synthesis machine learningmodel 405 may be configured to perform a plurality of steps/operationswith respect to an audio-based feature data object 408 and anacceleration-based feature data object 406 in order to generate ahybrid-input prediction data object 411. The feature synthesis machinelearning model 405 may comprise one or more layers configured to extractsequential information/understanding (e.g., recognize patterns) inhistorical data contained in the preceding machine learning model(s)and/or layers. A hybrid-input prediction data object 411 may refer to adata object that describes the output of two or more input data objects(e.g., an audio-based feature data object 408 and an acceleration-basedfeature data object 406) generated (e.g., merged, combined and/or thelike) by a feature synthesis machine learning model 405.

FIG. 7 provides an example system 700 for generating a hybrid-inputprediction data object 411 by a feature synthesis machine learning model405. As shown, the feature synthesis machine learning model 405comprises a plurality of layers configured to process two input dataobjects (e.g., the audio-based feature data object 408 and theacceleration-based feature data object 406) in order to generate anoutput (e.g., hybrid-input prediction data object 411) that integratesfeatures described by both of the input data objects.

As illustrated in FIG. 7 , the feature synthesis machine learning model405 comprises a concatenation layer 701 configured to process anacceleration-based feature data object 406 and an audio-based featuredata object 408 to generate a concatenation feature data object 712. Theconcatenation layer 701 may refer to a layer of the feature synthesismachine learning model 405 that is configured to combine (e.g., merge,stack and/or the like) two or more data objects (e.g., matrices) togenerate an output describing features in the two or more data objects,where the combination of the two or more data objects is intended toconcentrate the values of the two or more data objects along aconcatenation dimension (e.g., a horizontal dimension or a verticaldimension). For example, as shown, the concatenation layer 701 isconfigured to process the audio-based feature data object 408 and theacceleration-based feature data object 406 in order to generate aconcatenation feature data object 712. Concatenation of the audio-basedfeature data object 408 and the acceleration-based feature data object406 may include vector-by-vector steps/operations to extract newfeatures from the respective data objects. A concatenation feature dataobject 712 may refer to a data object that describes the output of theconcatenation layer 701.

As illustrated in FIG. 7 , the feature synthesis machine learning model405 comprises a synthesis model sequence modeling layer 703. Thesynthesis model sequence modeling layer 703 may refer to a layer of afeature synthesis machine learning model 405 that is configured toperform one or more synthesis model sequence steps/operations withrespect to an input data object (e.g., a concatenation feature dataobject 712) and generate a synthesis model sequence modeling output 714.For example, the synthesis model sequence modeling layer 703, mayfurther modify (e.g., flatten, combine) the input data object togenerate the synthesis model sequence modeling output 714. The synthesismodel sequence modeling layer 703 may comprise one or more gatedrecurrent units (GRUs), Recurrent Neural Networks (RNNs), LongShort-Term Memory (LSTM) units and/or the like. The synthesis modelsequence modeling output 714 may refer to a data object that describesthe output of one or more synthesis model sequence modeling layers 703of a feature synthesis machine learning model 405.

As illustrated in FIG. 7 , the feature synthesis machine learning model405 comprises a time-distributed fully connected layer 705 configured toprocess a synthesis model sequence modeling output 714 to generate ahybrid-input prediction data object 411. The time-distributed fullyconnected layer 705 may refer to a fully-connected layer of a featuresynthesis machine learning model 405 that is configured to perform oneor more prediction-based actions with respect to an input data object(e.g., a synthesis model sequence modeling output 714) and generate thehybrid-input prediction data object 411. An example of atime-distributed fully connected layer is the time-distributed Denselayer in the Keras application programming interface (API). Thetime-distributed fully connected layer may process an input data object(e.g., the synthesis model sequence modeling output 714) by extractingthe most relevant feature for each time segment to generate a hybridinput prediction data object. The time-distributed fully connected layer705 may be configured to determine the likelihood of one or more eventsbased at least in part on the hybrid-input prediction data object. Thetime-distributed fully connected layer may be configured to determine afrequency or duration corresponding with the one or more events acrossone or more time intervals. The hybrid-input prediction data object 411may refer to a data object that describes the output of thetime-distributed fully connected layer 705 of the feature synthesismachine learning model 405. An example hybrid-input prediction dataobject 411 may comprise a single vector comprising a single relevantfeature corresponding with each time segment.

FIG. 11 is a flowchart diagram illustrating an example process forgenerating a hybrid-input prediction data object 411 by a featuresynthesis machine learning model 405. The process depicted in FIG. 11begins at step/operation 1102, when the concatenation layer 701 of thefeature synthesis machine learning model 405 processes theacceleration-based feature data object 406 and the audio-based featuredata object 408 to generate a concatenation feature data object 712. Atstep/operation 1104, the synthesis model sequence modeling layer 703 ofthe feature synthesis machine learning model 405 processes theconcatenated feature data object to generate a synthesis model sequencemodeling output 714. Then at step/operation 1106, the time-distributedfully connected layer 705 processes the synthesis model sequencemodeling output 714 to generate the hybrid-input prediction data object411.

Performing Prediction-Based Actions

In various embodiments, the hybrid-input prediction system 101 may beconfigured to further perform or trigger prediction-based actions inresponse to generated predictive outputs (e.g., one or more generatedhybrid-input prediction data objects 411). For example, the hybrid-inputprediction system 101 may trigger generation (e.g., by a clientcomputing entity 102) of user interface data (e.g., messages, dataobjects and/or the like) for presentation by a user computing entity.

FIG. 13 provides an example graphical representation 1300 illustratingan acceleration-based feature data object 406, an audio-based featuredata object 408 and a hybrid-input prediction data object 411 generatedby a hybrid-input prediction system 101. As shown, theacceleration-based feature data object 406 comprises a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a number of segments in time and a second dimensioncorresponds with a plurality of features associated withoscillations/vibrations occurring at the corresponding segments in time.The audio-based feature data object 408 comprises a two-dimensionalmatrix having a length and a width in which a first dimensioncorresponds with a number of segments in time and a second dimensioncorresponds with a plurality of features associated with frequencies ofsound waves occurring at the corresponding segments in time. Thehybrid-input prediction data object 411 comprises a single vectorcomprising a single relevant feature corresponding with each timesegment. Each feature corresponds with a likelihood/probability of anevent at the corresponding time segment. As shown, the hybrid-inputprediction data object 411 indicates a very high likelihood of an eventoccurring at a single time segment (“1200”), based at least in part onthe data/features contained in the acceleration-based feature dataobject 406 and the audio-based feature data object 408.

FIG. 14 provides an operational example showing user interface datagenerated based at least in part on an alert and/or correspondingpredictive output (e.g., hybrid-input prediction data object 411). Thepredictive output may be provided (e.g., sent, transmitted and/or thelike) to a client computing entity 102. The client computing entity 102may generate a corresponding alert and provide (e.g., transmitted, sentand/or the like) corresponding user interface data for presentation by auser interface 1400. The user interface data may be used for dynamicallyupdating a user interface 1400. In some embodiments, the user interface1400 may dynamically update the display on a continuous or regular basisor in response to certain triggers. The user interface 1400 may comprisevarious features and functionality for accessing, and/or viewing userinterface data. In one embodiment, the user interface 1400 may identifythe user (e.g., monitored individual) credentialed for currentlyaccessing the user interface 1400 (e.g., John Doe). The user interface1400 may also comprise messages to the user in the form of banners,headers, notifications, and/or the like. As will be recognized, thedescribed elements are provided for illustrative purposes and are not tobe construed as limiting the dynamically updatable interface in any way.

VI. Conclusion

Many modifications and other embodiments will come to mind to oneskilled in the art to which this disclosure pertains having the benefitof the teachings presented in the foregoing descriptions and theassociated drawings. Therefore, it is to be understood that thedisclosure is not to be limited to the specific embodiments disclosedand that modifications and other embodiments are intended to be includedwithin the scope of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

The invention claimed is:
 1. A computer-implemented method comprising:generating, by one or more processors and using an audio processingmachine learning model and an audio data object, an audio-based featuredata object, wherein: the audio processing machine learning modelcomprises (i) an audio model layer that is configured to generate anaudio model output based at least in part on the audio data object, and(ii) an audio model one-dimensional convolutional layer that isconfigured to generate an audio model convolutional output based atleast in part on the audio model output, and the audio-based featuredata object is generated based at least in part on the audio modelconvolutional output; generating, by the one or more processors andusing an acceleration processing machine learning model and anacceleration data object, an acceleration-based feature data object,wherein: the acceleration processing machine learning model comprises(i) an acceleration model layer that is configured to generate anacceleration model output based at least in part on the accelerationdata object, (ii) an acceleration model one-dimensional convolutionallayer that is configured to generate an acceleration model convolutionaloutput based at least in part on the acceleration model output, and(iii) an acceleration model up-sampling layer that is configured togenerate an acceleration model up-sampling output based at least in parton the acceleration model convolutional output, and theacceleration-based feature data object is generated based at least inpart on the acceleration model up-sampling output; generating, by theone or more processors and using a feature synthesis machine learningmodel and the audio-based feature data object and the acceleration-basedfeature data object, a hybrid-input prediction data object; andinitiating, by the one or more processors, the performance of one ormore prediction-based actions based at least in part on the hybrid-inputprediction data object.
 2. The computer-implemented method of claim 1,wherein: the audio-based feature data object has an audio feature objecttarget dimension length, the acceleration-based feature data object hasan acceleration feature object target dimension length, and the audiofeature object target dimension length and the acceleration featureobject target dimension length are equal.
 3. The computer-implementedmethod of claim 2, wherein the acceleration processing machine learningmodel comprises an acceleration model dimension adjustment layerconfigured to: generate an acceleration model dimension adjustmentoutput based at least in part on the acceleration model up-samplingoutput by adjusting a target dimension of the acceleration modelup-sampling output in accordance with the acceleration feature objecttarget dimension length, and generate the audio-based feature dataobject based at least in part on the acceleration model dimensionadjustment output.
 4. The computer-implemented method of claim 3,wherein the acceleration model dimension adjustment layer comprises: anacceleration model sequence modeling layer that is configured togenerate an acceleration model sequence modeling output based at leastin part on the acceleration model up-sampling output, a secondacceleration model up-sampling layer that is configured to generate asecond acceleration model up-sampling output based at least in part onthe acceleration model sequence modeling output, and an accelerationmodel dimension truncation layer that is configured to generate theacceleration model dimension adjustment output based at least in part onthe second acceleration model up-sampling output.
 5. Thecomputer-implemented method of claim 1, wherein the feature synthesismachine learning model comprises: a concatenation layer that isconfigured to generate a concatenated feature data object based at leastin part on the audio-based feature data object and theacceleration-based feature data object, one or more synthesis modelsequence modeling layers that are collectively configured to generate asynthesis model sequence modeling output based at least in part on theconcatenated feature data object, and a time-distributed fully connectedlayer that is configured to generate the hybrid-input prediction dataobject based at least in part on the synthesis model sequence modelingoutput.
 6. The computer-implemented method of claim 1, wherein: theaudio data object describes recorded audio data for a monitoredindividual, and the acceleration data object describes recordedcross-body acceleration data for the monitored individual.
 7. Thecomputer-implemented method of claim 6, wherein the hybrid-inputprediction data object describes cough likelihoods for the monitoredindividual across one or more time intervals.
 8. Thecomputer-implemented method of claim 6, wherein the hybrid-inputprediction data object describes cough frequency for the monitoredindividual across one or more time intervals.
 9. Thecomputer-implemented method of claim 6, wherein the hybrid-inputprediction data object describes cough durations for the monitoredindividual across one or more time intervals.
 10. An apparatuscomprising at least one processor and at least one memory includingprogram code, the at least one memory and the program code configuredto, with the at least one processor, cause the apparatus to at least:generate using an audio processing machine learning model and an audiodata object, an audio-based feature data object, wherein: the audioprocessing machine learning model comprises (i) an audio model layerthat is configured to generate an audio model output based at least inpart on the audio data object, and (ii) an audio model one-dimensionalconvolutional layer that is configured to generate an audio modelconvolutional output based at least in part on the audio model output,and the audio-based feature data object is generated based at least inpart on the audio model convolutional output; generate using anacceleration processing machine learning model and an acceleration dataobject, an acceleration-based feature data object, wherein: theacceleration processing machine learning model comprises (i) anacceleration model layer that is configured to generate an accelerationmodel output based at least in part on the acceleration data object,(ii) an acceleration model one-dimensional convolutional layer that isconfigured to generate an acceleration model convolutional output basedat least in part on the acceleration model output, and (iii) anacceleration model up-sampling layer that is configured to generate anacceleration model up-sampling output based at least in part on theacceleration model convolutional output, and the acceleration-basedfeature data object is generated based at least in part on theacceleration model up-sampling output; generate using a featuresynthesis machine learning model and the audio-based feature data objectand the acceleration-based feature data object, a hybrid-inputprediction data object; and initiate the performance of one or moreprediction-based actions based at least in part on the hybrid-inputprediction data object.
 11. The apparatus of claim 10, wherein: theaudio-based feature data object has an audio feature object targetdimension length, the acceleration-based feature data object has anacceleration feature object target dimension length, and the audiofeature object target dimension length and the acceleration featureobject target dimension length are equal.
 12. The apparatus of claim 11,wherein the acceleration processing machine learning model comprises anacceleration model dimension adjustment layer, and the program code isfurther configured to, with the at least one processor, cause theapparatus to at least: generate an acceleration model dimensionadjustment output based at least in part on the acceleration modelup-sampling output by adjusting a target dimension of the accelerationmodel up-sampling output in accordance with the acceleration featureobject target dimension length, and generate the audio-based featuredata object based at least in part on the acceleration model dimensionadjustment output.
 13. The apparatus of claim 12, wherein theacceleration model dimension adjustment layer comprises: an accelerationmodel sequence modeling layer that is configured to generate anacceleration model sequence modeling output based at least in part onthe acceleration model up-sampling output, a second acceleration modelup-sampling layer that is configured to generate a second accelerationmodel up-sampling output based at least in part on the accelerationmodel sequence modeling output, and an acceleration model dimensiontruncation layer that is configured to generate the acceleration modeldimension adjustment output based at least in part on the secondacceleration model up-sampling output.
 14. The apparatus of claim 10,wherein the feature synthesis machine learning model comprises: aconcatenation layer that is configured to generate a concatenatedfeature data object based at least in part on the audio-based featuredata object and the acceleration-based feature data object, one or moresynthesis model sequence modeling layers that are collectivelyconfigured to generate a synthesis model sequence modeling output basedat least in part on the concatenated feature data object, and atime-distributed fully connected layer that is configured to generatethe hybrid-input prediction data object based at least in part on thesynthesis model sequence modeling output.
 15. The apparatus of claim 10,wherein: the audio data object describes recorded audio data for amonitored individual, and the acceleration data object describesrecorded cross-body acceleration data for the monitored individual. 16.A computer program product comprising at least one non-transitorycomputer-readable storage medium having computer-readable program codeportions stored therein, the computer-readable program code portionsconfigured to: generate using an audio processing machine learning modeland an audio data object, an audio-based feature data object, wherein:the audio processing machine learning model comprises (i) an audio modellayer that is configured to generate an audio model output based atleast in part on the audio data object, and (ii) an audio modelone-dimensional convolutional layer that is configured to generate anaudio model convolutional output based at least in part on the audiomodel output, and the audio-based feature data object is generated basedat least in part on the audio model convolutional output; generate usingan acceleration processing machine learning model and an accelerationdata object, an acceleration-based feature data object, wherein: theacceleration processing machine learning model comprises (i) anacceleration model layer that is configured to generate an accelerationmodel output based at least in part on the acceleration data object,(ii) an acceleration model one-dimensional convolutional layer that isconfigured to generate an acceleration model convolutional output basedat least in part on the acceleration model output, and (iii) anacceleration model up-sampling layer that is configured to generate anacceleration model up-sampling output based at least in part on theacceleration model convolutional output, and the acceleration-basedfeature data object is generated based at least in part on theacceleration model up-sampling output; generate using a featuresynthesis machine learning model and the audio-based feature data objectand the acceleration-based feature data object, a hybrid-inputprediction data object; and initiate the performance of one or moreprediction-based actions based at least in part on the hybrid-inputprediction data object.
 17. The computer program product of claim 16,wherein: the audio-based feature data object has an audio feature objecttarget dimension length, the acceleration-based feature data object hasan acceleration feature object target dimension length, and the audiofeature object target dimension length and the acceleration featureobject target dimension length are equal.
 18. The computer programproduct of claim 17, wherein the acceleration processing machinelearning model comprises an acceleration model dimension adjustmentlayer, and the computer-readable program code portions are furtherconfigured to: generate an acceleration model dimension adjustmentoutput based at least in part on the acceleration model up-samplingoutput by adjusting a target dimension of the acceleration modelup-sampling output in accordance with the acceleration feature objecttarget dimension length, and generate the audio-based feature dataobject based at least in part on the acceleration model dimensionadjustment output.
 19. The computer program product of claim 18, whereinthe acceleration model dimension adjustment layer comprises: anacceleration model sequence modeling layer that is configured togenerate an acceleration model sequence modeling output based at leastin part on the acceleration model up-sampling output, a secondacceleration model up-sampling layer that is configured to generate asecond acceleration model up-sampling output based at least in part onthe acceleration model sequence modeling output, and an accelerationmodel dimension truncation layer that is configured to generate theacceleration model dimension adjustment output based at least in part onthe second acceleration model up-sampling output.
 20. The computerprogram product of claim 16, wherein the feature synthesis machinelearning model comprises: a concatenation layer that is configured togenerate a concatenated feature data object based at least in part onthe audio-based feature data object and the acceleration-based featuredata object, one or more synthesis model sequence modeling layers thatare collectively configured to generate a synthesis model sequencemodeling output based at least in part on the concatenated feature dataobject, and a time-distributed fully connected layer that is configuredto generate the hybrid-input prediction data object based at least inpart on the synthesis model sequence modeling output.